docs/examples/Make Test Dataset.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "78e3457b",
   "metadata": {},
   "source": [
    "# Make Test Dataset\n",
    "\n",
    "When you encounter unexpected errors in the `power-grid-model`, you would like certainly to report the issue and debug (maybe by another developer) the calculation core with certain dataset. To make this possible, we have implemented a generic mechanism to export/import the dataset to/from JSON files, and to debug the calculation core in both Python and C++ with the test dataset. \n",
    "\n",
    "In this notebook we will learn how test datasets are made in this repository, including:\n",
    "\n",
    "* Structure of validation test datasets in this repository\n",
    "* Format of test datasets (JSON)\n",
    "* Use of helper functions to save and load the datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "05a5bd6e",
   "metadata": {},
   "source": [
    "## Structure of Validation Datasets\n",
    "\n",
    "All validation test datasets are located in the [tests/data](../../tests/data) folder. The structure of the folder is as follows:\n",
    "\n",
    "```\n",
    "data\n",
    "   |\n",
    "   |\n",
    "   - power_flow\n",
    "             |\n",
    "             - power_flow_testset_1\n",
    "             - power_flow_testset_2\n",
    "             - ...\n",
    "   - state_estimation\n",
    "             |\n",
    "             - state_estimation_testset_1\n",
    "             - state_estimation_testset_2\n",
    "             - ...\n",
    "```\n",
    "\n",
    "The testsets are separated in two types of calculations: `power_flow` and `state_estimation`. In each folder there are subfolders for individual testset. The test datasets are used in both [Python](../../tests/unit/test_0Z_model_validation.py) and [C++](../../tests/cpp_unit_tests/test_validation.cpp) unit tests. Therefore, once you create extra test datasets in the folder, you can debug the program in both Python and C++."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4315ddc8",
   "metadata": {},
   "source": [
    "### Individual Dataset\n",
    "\n",
    "An individual dataset folder (in either `power_flow` or `state_estimation`) will consists of following files:\n",
    "\n",
    "* `params.json`: calculation parameters, mandatory\n",
    "* `input.json`: input data, mandatory\n",
    "* `sym_output.json`: reference symmetric output\n",
    "* `asym_output.json`: reference asymmetric output\n",
    "* `update_batch.json`: update batch data, mandatory if `sym_output_batch.json` or `asym_output_batch.json` exists.\n",
    "* `sym_output_batch.json`: reference symmetric batch output\n",
    "* `asym_output_batch.json`: reference asymmetric batch output\n",
    "\n",
    "The `params.json` and `input.json` are always needed. The test program (in Python and C++) will detect other files and instantiate relevant test calculations. For example, if `sym_output.json` exists, the test program will run a one-time symmetric calculation and compare the results to the reference results in `sym_output.json`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e5460d14",
   "metadata": {},
   "source": [
    "#### Test Parameters\n",
    "\n",
    "The `params.json` looks something like this:\n",
    "\n",
    "```json\n",
    "{\n",
    "  \"calculation_method\": \"iterative_linear\",\n",
    "  \"rtol\": 1e-8,\n",
    "  \"atol\": {\n",
    "    \"default\": 1e-8,\n",
    "    \".+_residual\": 1e-4\n",
    "  },\n",
    "  \"independent\": false,\n",
    "  \"cache_topology\": true\n",
    "}\n",
    "```\n",
    "\n",
    "You need to specify the method to use for the calculation, the relative and absolute tolerance to compare the calculation results with the reference results. For `rtol` you always give one number. For `atol` you can also give one number, or you can give a dictionary with regular expressions to match the attribute names. In this way you can have fine control of individual tolerance for each attribut (e.g. active/reactive power). In the example it has an absolute tolerance of `1e-4` for attributes which ends with `_residual` and `1e-8` for everything else.\n",
    "\n",
    "The `independent` and `cache_topology` are needed for batch test cases. They specify if the update batch is indepedent and/or topology cachable. The test program will assert if the dataset indeed satisfy the specifications. See [Power Flow Example](./Power%20Flow%20Example.ipynb) for detailed explanation.\n",
    "\n",
    "The `calculation_method` can be one string or list of strings. In the latter case, the test program will run the validation test mutilple times using all the specified methods."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6b785f40",
   "metadata": {},
   "source": [
    "## JSON Data Format\n",
    "\n",
    "The data format for json is object based. You give a dictionary of all component types. Each entry is a list of components. And each component is a dictionary of named attributes and values. \n",
    "\n",
    "### Input\n",
    "\n",
    "See below as an example of `input.json`. Note that you don't need to specify the optional attributes if you don't need them. In the example the `r0` are `x0` are not specified for lines.\n",
    "\n",
    "```json\n",
    "{\n",
    "  \"node\": [\n",
    "    {\n",
    "      \"id\": 1,\n",
    "      \"u_rated\": 10.0\n",
    "    },\n",
    "    {\n",
    "      \"id\": 2,\n",
    "      \"u_rated\": 10.0\n",
    "    }\n",
    "  ],\n",
    "  \"line\": [\n",
    "    {\n",
    "      \"id\": 4,\n",
    "      \"from_node\": 1,\n",
    "      \"to_node\": 2,\n",
    "      \"from_status\": 1,\n",
    "      \"to_status\": 1,\n",
    "      \"r1\": 0.0,\n",
    "      \"x1\": 1.0,\n",
    "      \"c1\": 0.0,\n",
    "      \"tan1\": 0.0,\n",
    "      \"i_n\": 1e3\n",
    "    }\n",
    "  ],\n",
    "  \"source\": [\n",
    "    {\n",
    "      \"id\": 6,\n",
    "      \"node\": 1,\n",
    "      \"status\": 1,\n",
    "      \"u_ref\": 1.0,\n",
    "      \"sk\": 100.0,\n",
    "      \"rx_ratio\": 0.0\n",
    "    }\n",
    "  ],\n",
    "  \"sym_load\": [\n",
    "    {\n",
    "      \"id\": 7,\n",
    "      \"node\": 2,\n",
    "      \"status\": 1,\n",
    "      \"type\": 0,\n",
    "      \"p_specified\": 0.0,\n",
    "      \"q_specified\": 0.0\n",
    "    },\n",
    "    {\n",
    "      \"id\": 8,\n",
    "      \"node\": 2,\n",
    "      \"status\": 1,\n",
    "      \"type\": 1,\n",
    "      \"p_specified\": 0.0,\n",
    "      \"q_specified\": 0.0\n",
    "    }\n",
    "  ]\n",
    "}\n",
    "```\n",
    "\n",
    "### Output\n",
    "\n",
    "An example of `sym_output.json` is shown below. The test program will only compare the results which exist in the output file. For the example below, the program only compares the attribute of `id` and `u` for component `node`. Other results are not compared (e.g. `u_pu` for `node` or all attributes from `line`).\n",
    "\n",
    "```json\n",
    "{\n",
    "  \"node\": [\n",
    "    {\n",
    "      \"id\": 1,\n",
    "      \"u\": 10.0\n",
    "    },\n",
    "    {\n",
    "      \"id\": 2,\n",
    "      \"u\": 10.0\n",
    "    }\n",
    "  ]\n",
    "}\n",
    "```\n",
    "\n",
    "**It is important that you specify all the components of the same type (you can skip a type completely) in the output file, also in the same order as the input file!. The available attributes have also to be the same for all the components. the The program will fail if you specify the output in the following JSON files.**\n",
    "\n",
    "**Not all components are present for a type**\n",
    "```json\n",
    "{\n",
    "  \"node\": [\n",
    "    {\n",
    "      \"id\": 1,\n",
    "      \"u\": 10.0\n",
    "    }\n",
    "  ]\n",
    "}\n",
    "```\n",
    "\n",
    "**Order of the components are not the same as the input file**\n",
    "```json\n",
    "{\n",
    "  \"node\": [\n",
    "    {\n",
    "      \"id\": 2,\n",
    "      \"u\": 10.0\n",
    "    },\n",
    "    {\n",
    "      \"id\": 1,\n",
    "      \"u\": 10.0\n",
    "    }\n",
    "  ]\n",
    "}\n",
    "```\n",
    "\n",
    "**Attributes are not the same.**\n",
    "```json\n",
    "{\n",
    "  \"node\": [\n",
    "    {\n",
    "      \"id\": 1,\n",
    "      \"u_pu\": 1.0\n",
    "    },\n",
    "    {\n",
    "      \"id\": 2,\n",
    "      \"u\": 10.0\n",
    "    }\n",
    "  ]\n",
    "}\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f8b8a94c",
   "metadata": {},
   "source": [
    "### Batch Dataset\n",
    "\n",
    "You can also execute batch calculations in the test program. You need to then specify a series of mutations in `update_batch.json` in a list. See below an example. Each mutation is always applied to the original input dataset. You don't need to specify the same components in each batch.\n",
    "```json\n",
    "[\n",
    "  {\n",
    "    \"sym_load\": [\n",
    "      {\n",
    "        \"id\": 7,\n",
    "        \"q_specified\": 6.9444444444444455\n",
    "      }\n",
    "    ]\n",
    "  },\n",
    "  {\n",
    "    \"sym_load\": [\n",
    "      {\n",
    "        \"id\": 8,\n",
    "        \"q_specified\": 6.666666666666667\n",
    "      }\n",
    "    ]\n",
    "  },\n",
    "  {\n",
    "    \"sym_load\": [\n",
    "      {\n",
    "        \"id\": 7,\n",
    "        \"q_specified\": 2.132231404958678\n",
    "      },\n",
    "      {\n",
    "        \"id\": 8,\n",
    "        \"q_specified\": 2.4199999999999995\n",
    "      }\n",
    "    ]\n",
    "  }\n",
    "]\n",
    "```\n",
    "\n",
    "With the update dataset, you can then put a batch reference result file `sym_output_batch.json`:\n",
    "```json\n",
    "[\n",
    "  {\n",
    "    \"node\": [\n",
    "      {\n",
    "        \"id\": 1,\n",
    "        \"u\": 9.166666666666666\n",
    "      },\n",
    "      {\n",
    "        \"id\": 2,\n",
    "        \"u\": 8.333333333333334\n",
    "      }\n",
    "    ]\n",
    "  },\n",
    "  {\n",
    "    \"node\": [\n",
    "      {\n",
    "        \"id\": 1,\n",
    "        \"u\": 9.411764705882353\n",
    "      },\n",
    "      {\n",
    "        \"id\": 2,\n",
    "        \"u\": 8.823529411764707\n",
    "      }\n",
    "    ]\n",
    "  },\n",
    "  {\n",
    "    \"node\": [\n",
    "      {\n",
    "        \"id\": 1,\n",
    "        \"u\": 9.545454545454545\n",
    "      },\n",
    "      {\n",
    "        \"id\": 2,\n",
    "        \"u\": 9.090909090909092\n",
    "      }\n",
    "    ]\n",
    "  }\n",
    "]\n",
    "```\n",
    "\n",
    "**The length of the result list should be the same as the length of the update batch list.**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef0e663c",
   "metadata": {},
   "source": [
    "### Empty Result File\n",
    "\n",
    "If you encounter a crash for a certain dataset. You can also create the input data into JSON files. In this case you might not have any reference result to compare, because you just need to find where the crash happens. You still need an empty (dictionary) result file to trigger the calculation.\n",
    "\n",
    "For `sym_output.json` it is just an empty dictionary:\n",
    "```json\n",
    "{}\n",
    "```\n",
    "\n",
    "For `sym_output_batch.json` it is a list of empty dictionaries:\n",
    "```json\n",
    "[{}, {}, {}]\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "99220dfe",
   "metadata": {},
   "source": [
    "## Helper Functions to Import and Export\n",
    "\n",
    "In the module `power_grid_model.utils` we have some helper functions to import a json file to a `power-grid-model` compatible dataset, or the other way around. \n",
    "\n",
    "Please refer to the [source code](../../src/power_grid_model/utils.py) for detailed function signature.\n",
    "\n",
    "In this notebook we export the example network from [Power Flow](./Power%20Flow%20Example.ipynb) to json. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "b158e92f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# first build the network\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "from power_grid_model import LoadGenType\n",
    "from power_grid_model import PowerGridModel, CalculationMethod\n",
    "from power_grid_model import initialize_array\n",
    "\n",
    "# network\n",
    "\n",
    "# node\n",
    "node = initialize_array(\"input\", \"node\", 3)\n",
    "node[\"id\"] = [1, 2, 6]\n",
    "node[\"u_rated\"] = [10.5e3, 10.5e3, 10.5e3]\n",
    "\n",
    "# line\n",
    "line = initialize_array(\"input\", \"line\", 3)\n",
    "line[\"id\"] = [3, 5, 8]\n",
    "line[\"from_node\"] = [1, 2, 1]\n",
    "line[\"to_node\"] = [2, 6, 6]\n",
    "line[\"from_status\"] = [1, 1, 1]\n",
    "line[\"to_status\"] = [1, 1, 1]\n",
    "line[\"r1\"] = [0.25, 0.25, 0.25]\n",
    "line[\"x1\"] = [0.2, 0.2, 0.2]\n",
    "line[\"c1\"] = [10e-6, 10e-6, 10e-6]\n",
    "line[\"tan1\"] = [0.0, 0.0, 0.0]\n",
    "line[\"i_n\"] = [1000, 1000, 1000]\n",
    "\n",
    "# load\n",
    "sym_load = initialize_array(\"input\", \"sym_load\", 2)\n",
    "sym_load[\"id\"] = [4, 7]\n",
    "sym_load[\"node\"] = [2, 6]\n",
    "sym_load[\"status\"] = [1, 1]\n",
    "sym_load[\"type\"] = [LoadGenType.const_power, LoadGenType.const_power]\n",
    "sym_load[\"p_specified\"] = [20e6, 10e6]\n",
    "sym_load[\"q_specified\"] = [5e6, 2e6]\n",
    "\n",
    "# source\n",
    "source = initialize_array(\"input\", \"source\", 1)\n",
    "source[\"id\"] = [10]\n",
    "source[\"node\"] = [1]\n",
    "source[\"status\"] = [1]\n",
    "source[\"u_ref\"] = [1.0]\n",
    "\n",
    "# all\n",
    "input_data = {\"node\": node, \"line\": line, \"sym_load\": sym_load, \"source\": source}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "73dba42b",
   "metadata": {},
   "source": [
    "### Export to JSON\n",
    "\n",
    "We can use the fuction `export_json_data` the input data to a json file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "724e098a",
   "metadata": {},
   "outputs": [],
   "source": [
    "from power_grid_model.utils import export_json_data\n",
    "import tempfile\n",
    "from pathlib import Path\n",
    "\n",
    "temp_path = Path(tempfile.gettempdir())\n",
    "export_json_data(temp_path / \"input.json\", input_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "071c790a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"node\": [\n",
      "    {\n",
      "      \"id\": 1,\n",
      "      \"u_rated\": 10500.0\n",
      "    },\n",
      "    {\n",
      "      \"id\": 2,\n",
      "      \"u_rated\": 10500.0\n",
      "    },\n",
      "    {\n",
      "      \"id\": 6,\n",
      "      \"u_rated\": 10500.0\n",
      "    }\n",
      "  ],\n",
      "  \"line\": [\n",
      "    {\n",
      "      \"id\": 3,\n",
      "      \"from_node\": 1,\n",
      "      \"to_node\": 2,\n",
      "      \"from_status\": 1,\n",
      "      \"to_status\": 1,\n",
      "      \"r1\": 0.25,\n",
      "      \"x1\": 0.2,\n",
      "      \"c1\": 1e-05,\n",
      "      \"tan1\": 0.0,\n",
      "      \"i_n\": 1000.0\n",
      "    },\n",
      "    {\n",
      "      \"id\": 5,\n",
      "      \"from_node\": 2,\n",
      "      \"to_node\": 6,\n",
      "      \"from_status\": 1,\n",
      "      \"to_status\": 1,\n",
      "      \"r1\": 0.25,\n",
      "      \"x1\": 0.2,\n",
      "      \"c1\": 1e-05,\n",
      "      \"tan1\": 0.0,\n",
      "      \"i_n\": 1000.0\n",
      "    },\n",
      "    {\n",
      "      \"id\": 8,\n",
      "      \"from_node\": 1,\n",
      "      \"to_node\": 6,\n",
      "      \"from_status\": 1,\n",
      "      \"to_status\": 1,\n",
      "      \"r1\": 0.25,\n",
      "      \"x1\": 0.2,\n",
      "      \"c1\": 1e-05,\n",
      "      \"tan1\": 0.0,\n",
      "      \"i_n\": 1000.0\n",
      "    }\n",
      "  ],\n",
      "  \"sym_load\": [\n",
      "    {\n",
      "      \"id\": 4,\n",
      "      \"node\": 2,\n",
      "      \"status\": 1,\n",
      "      \"type\": 0,\n",
      "      \"p_specified\": 20000000.0,\n",
      "      \"q_specified\": 5000000.0\n",
      "    },\n",
      "    {\n",
      "      \"id\": 7,\n",
      "      \"node\": 6,\n",
      "      \"status\": 1,\n",
      "      \"type\": 0,\n",
      "      \"p_specified\": 10000000.0,\n",
      "      \"q_specified\": 2000000.0\n",
      "    }\n",
      "  ],\n",
      "  \"source\": [\n",
      "    {\n",
      "      \"id\": 10,\n",
      "      \"node\": 1,\n",
      "      \"status\": 1,\n",
      "      \"u_ref\": 1.0\n",
      "    }\n",
      "  ]\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "# we can display the json file\n",
    "\n",
    "with open(temp_path / \"input.json\", \"r\") as f:\n",
    "    print(f.read())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e85c767",
   "metadata": {},
   "source": [
    "### Import JSON\n",
    "\n",
    "We can use the fuction `import_json_data` the input data to a json file. To import json you need to specify the type of the data, it is either `input`, `update`, `sym_output`, or `asym_output`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "c79d7216",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   id  energized      u_pu             u   u_angle\n",
      "0   1          1  0.998988  10489.375043 -0.003039\n",
      "1   2          1  0.952126   9997.325181 -0.026031\n",
      "2   6          1  0.962096  10102.012975 -0.021895\n"
     ]
    }
   ],
   "source": [
    "# round trip and run power flow\n",
    "\n",
    "from power_grid_model.utils import import_json_data\n",
    "\n",
    "imported_data = import_json_data(temp_path / \"input.json\", \"input\")\n",
    "\n",
    "pgm = PowerGridModel(imported_data)\n",
    "result = pgm.calculate_power_flow()\n",
    "\n",
    "import pandas as pd\n",
    "\n",
    "print(pd.DataFrame(result[\"node\"]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f2409710",
   "metadata": {},
   "source": [
    "## Import and Export Batch Update/Result Dataset\n",
    "\n",
    "You can use the same function to import and export batch update/result dataset for batch calculation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "37cd5ade",
   "metadata": {},
   "outputs": [],
   "source": [
    "# create batch set\n",
    "\n",
    "load_profile = initialize_array(\"update\", \"sym_load\", (3, 2))\n",
    "load_profile[\"id\"] = [[4, 7]]\n",
    "# this is a scale of load from 0% to 100%\n",
    "load_profile[\"p_specified\"] = [[30e6, 15e6]] * np.linspace(0, 1, 3).reshape(-1, 1)\n",
    "\n",
    "\n",
    "time_series_mutation = {\"sym_load\": load_profile}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "89011a10",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[\n",
      "  {\n",
      "    \"sym_load\": [\n",
      "      {\n",
      "        \"id\": 4,\n",
      "        \"p_specified\": 0.0\n",
      "      },\n",
      "      {\n",
      "        \"id\": 7,\n",
      "        \"p_specified\": 0.0\n",
      "      }\n",
      "    ]\n",
      "  },\n",
      "  {\n",
      "    \"sym_load\": [\n",
      "      {\n",
      "        \"id\": 4,\n",
      "        \"p_specified\": 15000000.0\n",
      "      },\n",
      "      {\n",
      "        \"id\": 7,\n",
      "        \"p_specified\": 7500000.0\n",
      "      }\n",
      "    ]\n",
      "  },\n",
      "  {\n",
      "    \"sym_load\": [\n",
      "      {\n",
      "        \"id\": 4,\n",
      "        \"p_specified\": 30000000.0\n",
      "      },\n",
      "      {\n",
      "        \"id\": 7,\n",
      "        \"p_specified\": 15000000.0\n",
      "      }\n",
      "    ]\n",
      "  }\n",
      "]\n"
     ]
    }
   ],
   "source": [
    "# export and print\n",
    "\n",
    "export_json_data(temp_path / \"update_batch.json\", time_series_mutation)\n",
    "\n",
    "with open(temp_path / \"update_batch.json\", \"r\") as f:\n",
    "    print(f.read())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "8c4ad876",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[       0.        0.]\n",
      " [15000000.  7500000.]\n",
      " [30000000. 15000000.]]\n"
     ]
    }
   ],
   "source": [
    "# import round trip, calculate\n",
    "\n",
    "imported_batch_update = import_json_data(temp_path / \"update_batch.json\", \"update\")\n",
    "\n",
    "batch_result = pgm.calculate_power_flow(update_data=imported_batch_update)\n",
    "\n",
    "print(batch_result[\"sym_load\"][\"p\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b57f48a0",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.9.13 ('pgm-dev')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  },
  "vscode": {
   "interpreter": {
    "hash": "39f035c5e58f53e973c1999fdd4393d6db2dd3f04fe05f524988f90f4bbdb576"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}